Histological reconstruction and automated image analysis

ABSTRACT

A method for automated image analysis of a biological specimen by histological reconstruction. Also provided is an automated cell image method for analyzing a biological specimen that has consecutively been stained by either an in situ hybridization (ISH) method, or an immunohistochemistry (IHC) method or a nucleic acid stain, and counterstained. The method couples composite images in an automated manner for processing and analysis. To identify structure in tissue that cannot be captured in a single field of view image or a single staining technique, the disclosure provides a method for histological reconstruction to analyze many fields of view on potentially many slides simultaneously.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation (and claims the benefit of priorityunder 35 USC 120) of U.S. application Ser. No. 10/081,714, filed Feb.20, 2002 and now U.S. Pat. No. 6,631,203, which is a continuation ofU.S. application Ser. No. 09/344,308, filed Jun. 24, 1999 and now U.S.Pat. No. 6,418,236, which claims the benefit of priority under 35 U.S.C.§119(e) of U.S. provisional Application No. 60/129,384, filed Apr. 13,1999. The aforementioned applications are explicitly incorporated hereinby reference in their entirety and for all purposes.

TECHNICAL FIELD

The invention relates generally to light microscopy and, moreparticularly, to automated techniques of analyzing cytochemical andimmunohistochemical staining.

BACKGROUND OF THE INVENTION

In current cellular imaging systems, the area of a stained slidecontaining a stained cellular specimen is scanned by automatedmicroscopy systems. The entire cellular specimen area of the slide isimaged with a series of field of view images. For further analysis, eachfield of view image must be separately analyzed to locate completecandidate objects of interest within the field of view image. Thisapproach may be acceptable for cellular objects or clusters that are notso large that they extend beyond the field of view width or length of animage. Often, for both automated and manual analysis, only a singlefield of view is analyzed for morphological characteristics, so that thecontext of the analysis is limited to that field of view on a singleslide.

SUMMARY OF THE INVENTION

A method for automated image analysis of a biological specimen isdisclosed. A biological specimen is stained and counterstained for aspecific marker or in the instance of immunohistochemistry or in situhybridization, the marker is detectably labeled. Such labels includeenzyme, radioisotopes, fluorescence or other labels well known in theart. The sample is then scanned at a plurality of positions by aphotoimaging system to acquire an image. Adjacent positions are thenused to reconstruct and provide a full picture image of the wholesample. A reconstructed whole sample image may then be further processedto identify coordinates that may have an object or area of interest.These coordinates are automatically selected for a candidate object ofinterest. In a preferred embodiment, a low magnification image of thecandidate objects of interest is automatically obtained. Preferably theimage is a color digital image. The candidate object of interest pixelsin each sample are automatically morphologically processed to identifyartifact pixels and the remaining candidate object of interest pixels inthe sample not identified as artifact pixels. The apparatus obtainingthe low magnification image is adjusted to a higher magnification, toacquire a higher magnification image of the sample, at the locationcoordinates corresponding to the low magnification image, for eachcandidate object or area of interest. Pixels of the higher magnificationimage in the first color space are automatically transformed to a secondcolor space to differentiate higher magnification candidate objects ofinterest pixels from background pixels and identify, at highermagnification, objects of interest from the candidate object of interestpixels in the second color space. Thus, the pathologist or techniciancan identify whether the candidate object of interest has beenspecifically stained for the marker of interest, or counterstained, orboth specifically stained and counterstained.

An automated cellular image method for analyzing a biological specimen,that has consecutively been stained with hematoxylin and eosin (H/E) onone tissue section and by one or several immunohistochemistry (IHC)and/or in situ hybridization (ISH) methods on parallel tissue sections,is also disclosed as a particular embodiment. To identify a structure intissue that cannot be captured in a single field of view image or asingle staining technique, the invention provides a method forhistological reconstruction to analyze many fields of view onpotentially many slides simultaneously. The method couples compositeimages in an automated manner for processing and analysis. A slide onwhich is mounted a biological specimen stained to identify structure ofinterest is supported on a motorized stage. An image of the biologicalspecimen is generated, digitized, and stored. As the viewing field ofthe objective lens is smaller than the entire biological specimen, ahistological reconstruction is made. These stored images of the entiretissue section may then be placed together in an order such that the H/Estained slide is paired with the immunohistochemistry slide so thatanalysis of the images may be performed simultaneously by thepathologist. The images may be superimposed or viewed as adjacentimages.

In one embodiment, the invention provides a method for automated imageanalysis of a biological specimen by providing a biological sample to beanalyzed, automatically scanning the sample at a plurality ofcoordinates, automatically obtaining an is image at each of thecoordinates, reconstructing an image of the sample from each individualimage to create a reconstructed image and processing the reconstructedimage to identify a candidate object or area of interest.

In another embodiment, the invention provides a method for histologicalreconstruction. In this method a sample of a biological specimen isdivided into a number of subsamples. Each subsample is processed with astain, counterstain, immunohistochemical technique, in situhybridization technique, or a combination thereof. Each sample is thenscanned and an image is obtained from each of the samples. The imagesare then reconstructed such that a first image is paired with aconsecutive-corresponding sample image for identification of objects orareas of interest.

In yet another embodiment, the invention provides a computer programresiding on a computer-readable medium, for automated image analysis ofa biological specimen. The computer program comprises instructions forcausing a computer to process a sample by scanning the sample at aplurality of coordinates, obtaining an image at each of the coordinates,reconstructing the sample from the individual images to create areconstructed sample, identifying a coordinate of a candidate object orarea of interest in the reconstructed image and acquiring an image atthe coordinate obtained from the reconstructed image.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of an apparatus for automated cell imageanalysis.

FIG. 2 is a block diagram of the apparatus shown in FIG. 1.

FIG. 3 is a plan view of the apparatus of FIG. 1 having the housingremoved.

FIG. 4 is a side view of a microscope subsystem of the apparatus of FIG.1.

FIG. 5 shows a slide carrier. FIG. 5 a is a top view of a slide carrierfor use levity the apparatus of FIG. 1. FIG. 5 b is a bottom view of theslide carrier of FIG. 5 a.

FIG. 6 shows views of an automated slide handling subsystem.

FIG. 6 a is a top view of an automated slide handling subsystem of theapparatus of FIG. 1. FIG. 6 b is a partial cross-sectional view of theautomated slide handling subsystem of FIG. 6 a taken on line A—A.

FIG. 7 a-7 d illustrate the output operation of the automated slidehandling subsystem.

FIG. 8 is a flow diagram of the procedure for automatically determininga scan area.

FIG. 9 is a block diagram of the microscope controller of FIG. 2

FIG. 10 shows a method of histological reconstruction.

DETAILED DESCRIPTION

Overview

An automated microscope for analyzing cellular specimens is shown inFIGS. 1 and 3 and in block diagram in FIG. 2. A motorized stage 38 maybe used to support a slide 70 (FIG. 5). On the slide is mounted abiological specimen that is typically stained to identify a structure ofinterest for analysis. A biological specimen is defined as a cellular oracellular sample of biological origin, which has been mounted on amicroscope slide. The biological specimen can be, for example, a tissuesection, a biological fluid specimen, for example, a blood fractioncytospun on a microscope slide or a cell suspension directly seeded on aslide.

At least one optical sensing array, such as an objective lens, 44 a islocated above the stage and a light source 48 is located beneath thestage. Light from the source illuminates the stage and slide so an imageof the biological specimen is generated by the optical sensing array,for example an objective lens. This image is stored in memory.Preferably the image is a digitized or digital image. As the viewingfield of the optical array is smaller than the entire biologicalspecimen, the stage is moved in one planar direction by a distance thatcorresponds to the length of the field of view in that direction. Theimage generated at that position may then be captured and stored. Theacquired image may be flipped along its centerline due to the opticalflipping of the original image. Movement of the stage and capture ofresulting image continue in the same direction until the end of thespecimen area of the slide is reached. At that time, the stage is movedin the other planar direction by a distance that corresponds to thelength of the field of view in that direction and another image isgenerated and stored. The slide is traversed or scanned in this manneruntil the entire specimen area of the slide has been viewed through theobjective lens. These stored images may then be placed together in theorder in which they were collected to generate a composite orreconstructed image of the cellular specimen. This composite image maythen be analyzed to detect a structure that extends across more than oneimage field or more than one slide for further analysis. Such analysismay result in the identification of a candidate object or area ofinterest in both the field of view as well as objects that overlap twoor more fields of view. In such instances, the system will automaticallydetermine the coordinates for these candidate objects and may obtainadditional images at various magnifications.

In existing cellular imaging systems, the area of the slide is scannedby automated microscopy systems to image the entire specimen area of theslide with,a series of field of view images in the manner describedabove. However, these systems require that each field of view image isseparately analyzed to locate is complete candidate objects of interestwithin the field of view image for further analysis. This approach isacceptable for cellular objects that are not so large that they extendbeyond the field of view width or length of an image. To identifystructure in tissue that cannot be captured in a single field of viewimage, the present system uses an analysis method to identify field ofview images that appear to contain part of a tissue structure to beanalyzed. Field of view images so identified that are adjacent to oneanother are then identified as containing the tissue structure that thestain, antibody, or probe was intended to identify. This portion of thecomposite image may then be viewed under a higher magnification powerfor additional detail. The automatic building of a composite specimenimage from the field of view images and the processing of the compositeimage or its constituent images to identify tissue structure thatextends beyond one field of view is unknown.

In addition, a problem with existing automated systems is the continuedneed for operator input to initially locate cell objects for analysis.Such continued dependence on manual input can lead to errors includingobjects of interest being missed. These errors can be criticalespecially in assays for so-called rare events, e.g., finding onestained cell in a cell population of one million normal cells.Additionally, manual methods can be extremely time consuming and canrequire a high degree of training to properly identify or quantifycells. The associated manual labor leads to a high cost for theseprocedures in addition to the potential errors that can arise from long,tedious manual examinations. A need exists, therefore, for an improvedautomated cell image analysis system which can quickly and accuratelyscan large amounts of biological material on a slide.

Unless otherwise-defined, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention belongs. Although methods and materialssimilar or equivalent to those described herein can be used in thepractice or testing of the present invention, suitable methods andmaterials are described below.

Nuclear Stains, Intercalating Dyes and Counterstains

The term “nuclear stain” refers to a cytochemical stain thatpreferentially stains the nuclei of eukaryotic cells. Many nuclearstains are intercalating dyes. The term “intercalating dye” refers to achemical compound that can insert itself in between adjacent nucleotidesof a nucleic acid to provide a detectable color.

Many nuclear stains are known in the art, with one of the most commonlyused being hematoxylin. Hematoxylin is often used in combination withvarious metallic salts (mordants). Hematoxylin stains are used fordifferent staining purposes, and have a variety of colors, depending onthe metal used. Aluminum lakes are purple to blue, depending on pH. Ironlakes are blue-black. Chromium lakes are blue-black. Copper lakes areblue-green to purple. Nickel lakes are various shades of violet. Tinlakes are red. Lead lakes are dark brown. Osmium lakes are greenishbrown. Other nuclear stains include Giemsa stain, methyl green (whichbinds to AT-rich DNA regions), and Nuclear Fast-Red.

Fluorescent stains include Hoechst 33342; Hoechst 33258 (Calbiochem), abisbenzimide DNA intercalator that excites in the near UV (350 nm) andemits in the blue region (450 nm); thiazole orange, a fluorogenic stainfor DNA that excites in the blue region (515 nm) and emits in the greenregion (530 nm) of the visible spectrum; DAPI; ethidium bromide;propidium iodide; TOTO; YOYO-1; and SYTOX Blue or Green stains are alsoencompassed by the current invention. Several dyes either bind GC-richor AT-rich chromosomal regions preferentially or show differences influorescence intensity upon binding those regions, yielding fluorescentbanding patterns. For example, 7-Aminoactinomycin D binds selectively toGC-rich DNA regions and. 9-Amino-6-chloro-2-methoxyacridine fluoresceswith greatest intensity in AT-rich DNA regions. Acridine homodimerfluoresces preferentially when bound to AT-rich DNA regions.

The term “counterstain,” when used in combination with nuclear stains,refers to cytochemical stains that bind to a region of a eukaryotic cellother than the nucleus. Many counterstains are known in the art. One ofthe most common is eosin, which stains eukaryotic cell cytoplasm tovarying shades of pink. Other counterstains are specific for aparticular organelle or a protein in a cell. For example, theKleihauer-Betke cytochemical stain is specific for hemoglobin F, ahemoglobin type preferentially expressed in fetal cells and thereforecan be defined as a specific marker of fetal red blood cells. The term“coordinate” or “address” is used to mean a particular location on aslide or sample. The coordinate or address can be identified by anynumber of means including, for example, X-Y coordinates, r-Pcoordinates, and others recognized by those skilled in the art.

In one embodiment, an automated cellular imaging method is used toidentify fetal nucleated red blood cells in a maternal blood specimen.Fetal cells are first identified following the Kleihauer-Betkecytochemical stain for hemoglobin F. Fetal cells are identified by theautomated cellular imaging system as objects on the basis of theirbright red color (indicative of Hemoglobin F) as compared to-maternalred blood cells. To assure that appropriate objects are identified, sizeand shape morphological “filters” are used to exclude very small andvery large objects.

Cells are counterstained with an additional cytochemical stain fornucleic acids, resulting in a blue color for nucleated red blood cells(generally, fetal red blood cells). An automated image analysis systemidentifies blue objects of the appropriate size and shape for anerythrocyte nucleus among the bright red objects, allowing the imagingsystem to identify and enumerate nucleated fetal red cells. Such cellscan be enumerated, allowing for a screen for Down's syndrome in thefetus, wherein the frequency of such cells is typically higher in Down'ssyndrome pregnancies compared with normal pregnancies.

In a preferred embodiment, the slides are stained with hematoxylin/eosin(H/E) and one or several parallel slides containing adjacent sectionsare stained for one or several specific markers.

The results of the H/E staining provide cells with nuclei stainedblue-black, cytoplasm stained varying shades of pink; muscle fibersstained deep pinky red; fibrin stained deep pink; and red blood cellsstained orange-red.

For example, hematoxylin/eosin (H/E) slides are prepared with a standardH/E protocol. Standard solutions include the following: (1) Gillshematoxylin (hematoxylin 6.0 g; aluminum sulphate 4.2 g; citric acid 1.4g; sodium iodate 0.6 g; ethylene glycol 269 ml; distilled water 680 ml);(2) eosin (eosin yellowish 1.0 g; distilled water 100 ml); (3) lithiumcarbonate 1% (lithium carbonate 1 g; distilled water 100 g); (4) acidalcohol 1% 70% (alcohol 99 ml conc.; hydrochloric acid 1 ml); and (5)Scott's tap water. In a beaker containing 1 L distilled water, add 20 gsodium bicarbonate and 3.5 g magnesium sulphate. Add a magnetic stirrerand mix thoroughly to dissolve the salts. Using a filter funnel, pourthe solution into a labeled bottle.

The staining procedure is as follows: (1) Bring the tissue or cellsections to water; (2) place sections in hematoxylin for 5 minutes(min); (3) wash in tap water; (4) ‘blue’ the sections in lithiumcarbonate or Scott's tap water; (5) wash in tap water; (6) placesections in 1% acid alcohol for a few seconds; (7) wash in tap water;(8)place sections in eosin for 5 min; (9) wash in tap water; and (10)dehydrate with graded alcohol solution.

Mount Sections

A specific marker is a molecule or a group of molecules, which ispresent in only a subset of the components of a biological specimen andtherefore identifying specifically the components having the marker.Specific markers are frequently defined as antigens recognized byspecific antibodies (monoclonals or polyclonals) and can be detected byimmunohistochemistry.

Another group of specific markers is defined by the capacity of thesemarkers to hybridize, specifically, a nucleic acid probe. These markerscan usually be detected by in situ hybridization.

A third group of specific markers can be defined by their enzymaticactivity and can be detected by histochemistry.

A fourth group of specific markers can be stained directly,histochemically, using a specific dye.

A fifth group of specific markers can be defined as being receptorsbinding specifically to one or several ligands. A specific ligand isitself used for the detection of the receptor-ligand complex, using adetection method involving either histochemistry, orimmunohistochemistry or in situ hybridization.

Immunohistochemical and In Situ Hybridization Techniques

Immunohistochemical techniques as used herein encompasses the use ofreagents detecting cell specific markers, such reagents include, forexample, as antibodies and nucleic acids probes. Antibodies, includingmonoclonal antibodies, polyclonal antibodies and fragments thereof, areoften used to identify proteins or polypeptides of interest in a sample.A number of techniques are utilized to label objects of interestaccording to immunohistochemical techniques. Such techniques arediscussed in Current Protocols in Molecular Biology, Unit 14 et seq.,eds. Ausubel, et al., John Wiley & Sons, 1995, the disclosure of whichis incorporated herein by reference. For example, the followingprocedure is an example of immunohistochemical staining using anantibody recognizing, specifically, the HER2 protein. HER2overexpression has been described as a specific marker in a highpercentage of breast cancer carcinomas. This protocol is intended forstaining a paraffin embedded tissue section prepared according to aconventional procedure.

The section is deparaffinized using two baths of xylene and rehydratedthrough graded alcohols baths and finally in deionized water. Thesection is then incubated with an Antigen Retrieval Buffer, containingCitrate, for 40 minutes at 95° C. The slide is then cooled down at roomtemperature for 20 minutes in the same buffer. The slide is then rinsedin deionized water.

The area surrounding the tissue section is carefully dried and ahydrophobic delimiting pen is used to draw a line around the specimen,on the glass slide. A peroxidase blocking solution is added on thesection and incubated 5 minutes at room temperature. After being washedtwice with the Wash Buffer (a balanced salt solution), the tis,suesection is incubated 30 minutes at room temperature, with the primaryantibody recognizing the HER2 protein.

After 3 washes with the Wash Buffer, the tissue section is incubatedwith the peroxidase conjugated secondary antibody. The secondaryantibody will recognize specifically the primary antibody. The slide isthen washed 3 times with the Wash Buffer. Then the tissue section isincubated in presence of DAB and hydrogen peroxide for 10 minutes,before being washed with water.

The tissue section is counterstained in Hematoxylin for 2 minutes andrinsed again with water. The slide is mounted with a cover-slip using anaqueous mounting medium.

Immunohistochemical localization of cellular molecules uses the abilityof antibodies to bind specific antigens, for example proteins ofinterest such as onco-proteins and enzymes, with high affinity. Theseantibodies can be used to localize antigens to subcellular compartmentsor individual cells within a tissue.

In situ hybridization techniques include the use of specifically labelednucleic acid probes, which bind to cellular RNA or DNA in individualcells or tissue section. Suitable nucleic acid probes may be preparedusing standard molecular biology techniques including subcloning plasmidpreparation, and radiolabeling or non-radioactive labeling of thenucleic acid probe.

In situ hybridization is often performed on either paraffin or frozensections. Such techniques often include fine sectioning of tissues toprovide samples that are only a single to a few cell layers thick. Forexample paraffin blocks containing a tissue sample are cut into thin,approximately 8 um tissue sections, which are subsequently mounted onsubbed slides to be further processed for in situ hybridization.Alternatively, methacrylate may be used for sectioning. Cryosectioningtechniques are particularly suitable for immunohistochemistry and enzymehistochemistry.

Immunofluorescent labeling of a tissue section often use a sandwichassay or a primary antibody and secondary antibody-fluorochromeconjugate. Slides containing a tissue section of interest are washed inphosphate buffered saline and then exposed to a primary antibody whichwill bind to the protein object of interest. Subsequently the slides arewashed and exposed to the secondary-antibody which binds to the first orprimary antibody. The slide is washed and then developed. Numerous othertechniques well known in the art of immunohistochemical staining and insitu hybridization are easily adaptable for use in immunohistochemicalreconstruction as disclosed herein.

Thus, a combination of techniques using both chemical staining and/orimmunohistochemical and/or in situ hybridization may be used in thepresent methods. For example, numerous subsamples may be prepared from asingle tissue specimen. A first subsample may be chemically stained asdiscussed above, and a subsequent subsample may be subjected toimmunohistochemical and in situ hybridization techniques. Eachsubsamples is scanned and processed as discussed below. Reconstructedimages may then be overlapped or processed to further identify objectsof interest.

Histological Reconstruction

Histological reconstruction is a process whereby an image of a wholespecimen is constructed from analyzed pieces of the specimen,particularly when the biological specimen has been mounted on a slide.This image is created by piecing together more than one field of view atany particular magnification.

With reference to FIG. 10, an image 302, representing an objective'sfield of view is acquired at a first particular coordinate on the slidesample 301. The slide is automatically repositioned on the X-Y stage toobtain a new or second field of view corresponding to a secondparticular coordinate 302. This new field of view is preferablyimmediately adjacent to the first field of view, however, so long as thecoordinates, thus the address/identity, of each field of view areretained in the imaging system, histological reconstruction may beperformed. This process is repeated until images for the whole of thespecimen have been acquired.

Based upon each image's X and Y coordinate, the specimen is digitallyreconstructed. As part of the reconstruction, the image may be flippedto correct for the optical flipping of the original image.

The process of forming a histological reconstructed image involveshating the apparatus scan a microscope slide of interest, and form theimage that constitutes a reconstruction of the images taken during thescan. The image that is formed can be a full-color reconstruction of theentire scan area, or a fraction of the whole scan area, for example,reconstruction of the entire scan area that identifies objects or areasof interests. The reconstructed digital image can then be used forfurther processing or analysis to identify previously undetected objectsor areas of interest. For example, objects or areas of interestoverlapping one or more fields of view or slides may thus be identifiedin the reconstructed digital image.

With reference to FIGS. 1 and 2, the apparatus 10 also referred to asthe system, comprises a microscope 32 with a motorized X,Y and Z stage38, a camera 42, a computer 22 adapted to receive and process videoimages, and a set of software programs to control the apparatus and toexecute the method. A measurement of the optical properties of thesample features is used to form an image of the scannable area of theslide, to find sub-regions of interest, and to analyze the properties ofthese regions. The image processing method that evaluates the sample tofind regions of interest uses a measure of the hue, saturation orintensity and luminosity of a 24-bit color image to produce a white onblack target image of interest. This image is processed by separatelyconverting the full color image to components of hue, saturation orintensity and luminosity, thresholding the components, and performing alogical “AND” between the two images, then thresholding the resultingimage such that any pixel value above zero becomes 255. The processingand image acquisition will be further understood with reference to theapparatus described below.

Automated System

The invention provides a method for automated analysis of a biologicalspecimen, which eliminates the need for operator input to locatebiological objects or areas of interest for analysis.

With reference to FIG. 1, a slide prepared with a biological specimenand reagent is placed in a slide carrier 60 (FIG. 5) which preferablyholds four slides. The slide carriers are loaded into an input hopper 16of the automated system 10. The operator then enters data identifyingthe instrument protocol which contains information on the size, shapeand location of a scan area on each slide, or, preferably, the systemautomatically locates a scan area for each slide during slideprocessing. The operator then activates the system 10 for slideprocessing. At system activation, a slide carrier 60 is positioned on anX-Y stage 38 of an optical system, such as microscope subsystem 32. Anybar codes used to identify slides are read and stored for each slide inthe carrier. The entire slide is scanned at a low magnification,typically 10×. At each location of the scan, a low magnification imageis acquired and processed to detect candidate objects or areas ofinterest. Preferably, color, size and shape are used to identify objectsor areas of interest. The location of each candidate object or area ofinterest may be stored by reference to its coordinates or address. Eachfield of view may also be stored as part of a larger composite image(described in detail elsewhere).

At the completion of the low level scan for each slide in the carrier onthe stage, the optical system may be adjusted to a higher magnificationsuch as 40× or 60×, for additional specimen processing and imageacquisition, and the X-Y stage is positioned to the stored locations forthe candidate objects or areas of interest on each slide in the carrier.A higher magnification image is acquired for each candidate object orarea of interest and a series of image processing steps are performed toconfirm the analysis, which was performed at low magnification. A highermagnification image is stored for each continued object or area ofinterest. These images are then available for retrieval by a pathologistor cytotechnologist to review for final diagnostic evaluation. Havingstored the location of each object or area of interest, a mosaiccomprising the candidate objects or areas of interest for a slide may begenerated and stored. The pathologist or cytotechnologist may view themosaic or may also directly view the slide at the location of an objector area of interest in the mosaic for further evaluation. The mosaic maybe stored on magnetic or optical media for future reference or may betransmitted to a remote site for review or storage. The entire processinvolved in examining a single slide takes on the order of 4-100 mindepending on scan area size and the number of detected candidate objectsof interest.

The invention has utility in prenatal diagnosis of fetal cells, rareevent detection, rapid cell counting, tissue evaluation, and otherdiagnoses.

The processing of images acquired in the automated scanning preferablyincludes the steps of transforming the image to a different color space,here preferably by hue, saturation and intensity. The pixels of thefiltered image are dynamically thresholded to suppress backgroundmaterial; performing a morphological function to remove artifacts fromthe thresholded image; analyzing the thresholded image to determine thepresence of one or more regions of connected pixels having the samecolor; and categorizing every region having a size greater than aminimum size as a candidate object or area of interest.

According to another aspect, the scan area is automatically determinedby scanning the slide; acquiring an image at each slide position;analyzing texture or color information for each image to detect theedges of the specimen and storing the locations corresponding to thedetected edges to define the scan area.

According to yet another aspect, automated focusing of the opticalsystem is achieved by initially determining a focal surface from anarray of points or locations in the scan area. The derived focal surfaceenables subsequent rapid automatic focusing in the low power scanningoperation. In one embodiment, the focal plane is determined bydetermining proper focal positions across an array of locations andperforming a least squares fit of the array of focal positions to yielda focal plane across the array. Preferably, a focal position at eachlocation is determined by incrementing the position of a Z stage for afixed-number of coarse and fine iterations. At each iteration, an imageis acquired and a pixel variance, morphological gradient or otheroptical parameter about a pixel mean for the acquired image iscalculated to form a set of evaluation data. The peak value of the leastsquares fit curve is selected as an estimate of the best focal position.

In another aspect, a focal position method for a higher magnificationlocates a region of interest centered about a candidate object ofinterest within a slide which was located during an analysis of the lowmagnification images. The region of interest is preferably n columnswide, where n is a power of 2. The pixels of this region are thenprocessed using a Fast Fourier Transform to generate a spectra ofcomponent frequencies and corresponding complex magnitude for eachfrequency component. Preferably, the complex magnitude of the frequencycomponents which range from 25% to 75% of the maximum frequencycomponent are squared and summed to obtain the total power for theregion of interest. This process is repeated for other Z positions andthe Z position corresponding to the maximum total power for the regionof interest is selected as the best focal position. This focal methodcan be used with many stains and types of cellular specimens.

According to still another aspect, a method for automated slide handlingis provided. A slide is mounted onto a slide carrier 60 (FIG. 5) with anumber of other slides side-by-side. The slide carrier 60 is positionedin an input feeder 16 with other slide carriers to facilitate automaticanalysis off a batch of slides. The slide carrier is loaded onto the X-Ystage 38 of the optical system 32 for the analysis of the slidesthereon. Subsequently, the first slide carrier is unloaded into anoutput feeder 18 after automatic image analysis and the next carrier isautomatically loaded.

Referring to the FIGURES, an apparatus for automated cell image analysisof biological specimens is generally indicated by reference numeral 10as shown in perspective view in FIG. 1 and in block diagram form in FIG.2. The apparatus 10 comprises a microscope subsystem 32 housed in ahousing 12. The housing 12 includes a slide carrier input hopper 16 anda slide carrier output hopper 18. A door 14 in the housing 12 securesthe microscope subsystem from the external environment. A computersubsystem comprises a computer 22 having two system processors 23, animage processor 25 and a communications modem 29. The computer subsystemfurther includes a computer monitor 26 and an image monitor 27 and otherexternal peripherals including storage device 21, pointing device 30,keyboard 28 and color printer 35. An external power supply 24 is alsoshown for powering the system. Viewing oculars 20 of the microscopesubsystem project from the housing 12 for operator viewing. Theapparatus 10 further includes a 3-chip CCD camera 42 for acquiringimages through the microscope subsystem.32. The computer directlycontrols a number of microscope subsystem functions described further indetail.

An automatic slide feed mechanism 37 in conjunction with X-Y stage 38provide automatic slide handling in the apparatus 10. An illuminationlight source 48 projects light onto the X-Y stage 38 which issubsequently imaged through the microscope subsystem 32 and acquiredthrough the 3-chip CCD camera 42 for processing by the image processor25. A Z stage or focus stage 46 under control of the microscopecontroller 31 provides displacement of the microscope subsystem in the Zplane for focusing. The microscope subsystem 32 further includes amotorized objective turret 44 for selection of objectives.

The apparatus 10 is for the unattended automatic scanning of preparedmicroscope slides for the detection and counting of candidate objects orareas of interest, such as stained cells. In one embodiment theinvention may be utilized for tissue analysis. In another embodiment,rare event detection in which there may be only one candidate object ofinterest per several hundred thousand normal cells, e.g., one to fivecandidate objects of interest per 2 square centimeter area of the slide.The apparatus 10 automatically locates and counts candidate objects orareas of interest and estimates normal cells present in a cellularspecimen on the basis, for example, of color, size and shapecharacteristics. A biological specimen may be prepared with a reagent toobtain a colored insoluble precipitate. The apparatus, in oneembodiment, is used to detect this precipitate as a candidate object orarea of interest.

During operation of the apparatus 10, a pathologist or laboratorytechnician mounts prepared slides onto slide carriers. A slide carrier60 is illustrated in FIG. 5 and is described further below. Each slidecarrier holds up to 4 slides. Up to 25 slide carriers are then loadedinto input hopper 16. The operator can specify the size, shape andlocation of the area to be scanned or alternatively, the system canautomatically locate this area. The operator then commands the system tobegin automated scanning of the slides through a graphical userinterface. Unattended scanning begins with the automatic loading of thefirst carrier and slide onto the motorized X-Y stage 38. A bar codelabel affixed to the slide is read by a bar code reader 33 during thisloading operation. Each slide is then scanned at a user selected lowmicroscope magnification, for example, 10×, to build a histologicalreconstruction or identify candidate objects based on their color, sizeand shape characteristics. The X-Y locations of candidate objects orareas of interest are stored until scanning is completed.

After the low magnification scanning is completed, the apparatus mayautomatically return to each candidate object or area of interest, ifnecessary, reimaging and refocusing at a higher magnification such as40× and performs further analysis to confirm the biological candidate.The apparatus stores an image of the object or area of interest forlater review by pathologist. All results and images can be stored to astorage device 21 such as a removable hard drive or optical disc or DATtape or transmitted to a remote site for review or storage. The storedimages for each slide can be viewed in a mosaic of images for furtherreview. In addition, the pathologist or operator can also directly viewa detected object or area of interest through the microscope using theincluded oculars 20 or on image monitor 27.

The two system processors 102 further control an illumination controller106 for control of substage illumination 48. The light output from, forexample, a halogen light bulb, which supplies illumination for thesystem, can vary over time due to bulb aging, changes in opticalalignment, and other factors. In addition, slides which have been “overstained” can reduce the camera exposure to an unacceptable level. Tocompensate for these effects, the illumination controller 106 isincluded. This controller is used in conjunction with light controlsoftware to compensate for the variations in light level. The lightcontrol software samples the output from the camera at intervals (suchas between loading of slide carriers), and commands the controller toadjust the light level to the desired levels. In this way, light controlis automatic and transparent to the user and adds no additional time tosystem operation.

The system processors 23 are preferably comprised of dual parallel IntelPentium Pro 400 MHZ devices. The image processor 25 is preferably aMatrox Genesis board. The computer, in a preferred embodiment, operatesunder Windows NT. It will be recognized that any number of processorsand operating systems can be used in the methods and in conjunction withthe present invention.

Referring now to FIGS. 3 and 4, further detail of the apparatus 10 isshown. FIG. 3 shows a plan view of the apparatus 10 with the housing 12removed. A portion of the automatic slide feed mechanism 37 is shown tothe left of the microscope subsystem 32 and includes slide carrierunloading assembly 34 and unloading platform 36 which in conjunctionwith slide carrier unloading hopper 18 function to receive slidecarriers which have been analyzed.

Vibration isolation mounts 40, shown in further detail in FIG. 4, areprovided to isolate the microscope subsystem 32 from mechanical shockand vibration that can occur in a typical laboratory environment. Inaddition to external sources of vibration, the high speed operation ofthe X-Y stage 38 can induce vibration into the microscope subsystem 32.Such sources of vibration can be isolated from the electro-opticalsubsystems to avoid any undesirable effects on image quality. Theisolation mounts 40 comprise a spring 40 a and piston 40 b submerged ina high viscosity silicon gel which is enclosed in an elastomer membranebonded to a casing to achieve damping factors on the order of 17% to20%.

The automated slide handling subsystem operates on a single slidecarrier at a time. A slide carrier 60 is shown in FIGS. 5 a & 5 b, whichprovide a top view and a bottom view respectively. The slide carrier 60includes up to four slides 70 mounted with adhesive tape 62. The carrier60 includes ears 61 for hanging the carrier in the output hopper 18. Anundercut 66 and pitch rack 68 are formed at the top edge of the slidecarrier 60 for mechanical handling of the slide carrier. A keyway cutout65 is formed in one side of the carrier 60 to facilitate carrieralignment. A prepared slide 72 mounted on the slide carrier 60 includesa sample area 72 a and a bar code label area 72 b.

FIG. 6 a provides a top view of the slide handling subsystem whichcomprises a slide input module 15, a slide output module 17 and X-Ystage drive belt 50. FIG. 6 b provides a partial cross-sectional viewtaken along line A—A of FIG. 6 a.

The slide input module 15 includes a slide carrier input hopper 16,loading platform 52 and slide carrier loading subassembly 54. The inputhopper 16 receives a series of slide carriers 60 (FIGS. 5 a and 5 b) ina stack on loading platform 52. A guide key 57 protrudes from a side ofthe input hopper 16 to which the keyway cutout 65 (FIG. 5 a) of thecarrier is fit to achieve proper alignment.

The input module 15 further includes a revolving indexing cam 56 and aswitch 90 mounted in the loading platform 52, the operation of which isdescribed further below. The carrier subassembly 54 comprises an infeeddrive belt 59 driven by a motor 86. The infeed drive belt 59 includes apusher tab 58 for pushing the slide carrier horizontally toward the X-Ystage 38 when the belt is driven. A homing switch 95 senses the pushertab 58 during a revolution of the belt 59.

Referring specifically to FIG. 6 a, the X-Y stage 38 is shown with xposition and y position motors 96 and 97 respectively which arecontrolled by the microscope controller 31 (FIG. 9) and are notconsidered part of the slide handling subsystem. The X-Y stage 38further includes an aperture 55 for allowing illumination to reach theslide carrier. A switch 91 is mounted adjacent the aperture 55 forsensing contact with the carrier and thereupon activating a motor 87 todrive stage drive belt 50 (FIG. 6 b). The drive belt 50 is adouble-sided timing belt having teeth for engaging pitch rack 68 of thecarrier 60 (FIG. 5 b).

The slide output module 17 includes slide carrier output hopper 18,unloading platform 6, and slide carrier unloading subassembly 34. Theunloading subassembly 34 is a motor 89 for rotating the unloadingplatform 36 about shaft 98 during an unloading operation describedfurther below. An outfeed gear 93 driven by motor 88 rotatably engagesthe pitch rack 68 of the carrier 60 (FIG. 5 b) to transport the carrierto a rest position against switch 92. A spring loaded hold-downmechanism holds the carrier in place on the unloading platform 36.

The slide handling operation is now described. Referring to FIG. 7, aseries of slide carriers 60 are shown stacked in input hopper 16 withthe top edges 60 a aligned. As the slide handling operation begins, theindexing cam 56 driven by motor 85 advances one revolution to allow onlyone slide carrier to drop to the bottom of the hopper 16 and onto theloading platform 52.

FIGS. 7 a-7 d show the cam action in more detail. The indexing cam 56includes a hub 56 a to which are mounted upper and lower leaves 56 b and56 c respectively. The leaves 56 b & 56 c are semicircular projectionsoppositely positioned and spaced apart vertically. In a first positionshown in FIG. 8 a, the upper leaf 56 b supports the bottom carrier atthe undercut portion 66. At a position of the indexing cam 56 rotated180°, shown in FIG. 7 b, the upper leaf 56 b no longer supports thecarrier and instead the carrier has dropped slightly and is supported bythe lower leaf 56 c. FIG. 8 c shows the position of the cam 56 rotated270° wherein the upper leaf 56 b has rotated sufficiently to begin toengage the undercut 66 of the next slide carrier while the oppositefacing lower leaf 56 c still supports the bottom carrier. After a fullrotation of 360° as shown in FIG. 7 d, the lower leaf 56 c has rotatedopposite the carrier stack and no longer supports the bottom carrierwhich now rests on the loading platform 52. At the same position, theupper leaf 56 b supports the next carrier for repeating the cycle.

Referring again to FIGS. 6 a & 6 b, when the carrier drops to theloading platform 52, the contact closes switch 90 which activates motors86 and 87. Motor 86 drives the infeed drive belt 59 until the pusher tab58 makes contact with the carrier and pushes the carrier onto the X-Ystage drive belt 50. The stage drive belt 50 advances the carrier untilcontact is made with switch 91, the closing of which begins the slidescanning process described further herein. Upon completion of the Sscanning process, the X-Y stage 38 moves to an unload position andmotors 8, and 88 are activated to transport the carrier to the unloadingplatform 36 using stage drive belt 50. Motor 88 drives outfeed gear 93to engage the carrier pitch rack 68 of the carrier 60 (FIG. 5 b) untilswitch 92 is contacted. Closing switch 92 activates motor 89 to rotatethe unloading platform 36.

The unloading operation is shown in more detail in end views of theoutput module 17 (FIGS. 7 a-7 d). In FIG. 7 a, the unloading platform 36is shown in a horizontal position supporting a slide carrier 60. Thehold-down mechanism 94 secures the carrier 60 at one end. FIG. 7 b showsthe output module 17 after motor 89 has rotated the unloading platform36 to a vertical position, at which point the spring loaded hold-downmechanism 94 releases the slide carrier 60 into the output hopper 18.The carrier 60 is supported in the output hopper 18 by means of ears 64(FIGS. 5 a and 5 b). FIG. 7 c shows the unloading platform 16 beingrotated back towards the horizontal position. The platform 36 rotatesupward and contacts the deposited carrier 60. The upward movement pushesthe carrier toward the front of the output hopper. 18. FIG. 7 d showsthe unloading platform 36 at its original horizontal position afterhaving output a series of slide carriers 60 to the output hopper 18.

The aspects of the apparatus 10 relating to scanning, focusing and imageprocessing are further described in patent application Ser. No.08/758,436 which is incorporated herein.

Computer Implementation

Aspects of the invention may be implemented in hardware or software, ora combination of both. However, preferably, the algorithms and processesof the invention are implemented in one or more computer programsexecuting on programmable computers each comprising at least oneprocessor, at least one data storage system (including volatile andnon-volatile memory and/or storage elements), at least one input device,and at least one output device. Program code is applied to input data toperform the functions described herein and generate output information.The output information is applied to one or more output devices, inknown fashion.

Each program may be implemented in any desired computer language(including machine, assembly, high level procedural, or object orientedprogramming languages) to communicate with a computer system. In anycase, the language may be a compiled or interpreted language.

Each such computer program is preferably stored on a storage media ordevice (e.g., ROM, CD-ROM, tape, or magnetic diskette) readable by ageneral or special purpose programmable computer, for configuring andoperating the computer when the storage media or device is read by thecomputer to perform the procedures described herein. The inventivesystem may also be considered to be implemented as a computer-readablestorage medium, configured with a computer program, where the storagemedium so configured causes a computer to operate in a specific andpredefined manner to perform the functions described herein.

A number of embodiments of the present invention have been described.Nevertheless, various modifications may be made without departing fromthe spirit and scope of the invention. Accordingly, the invention is notto be limited by the specific illustrated embodiment, but only by thescope of the appended claims.

1. A method for automated image analysis of a plurality of biologicalsamples, comprising: (a) providing a plurality of biological samples tobe analyzed; (b) automatically scanning the plurality of biologicalsamples at a plurality of different positions with differentcoordinates; (c) automatically obtaining an image at each of saiddifferent coordinates to produce different images from differentlocations; (d) automatically reconstructing an image of the plurality ofsamples from the obtained different images comprising pairingconsecutive-corresponding images such that a reconstructed image of theplurality of samples is obtained; and (e) processing the reconstructedimage to automatically identify a candidate object or area of interest.2. The method of claim 1, wherein the reconstructed image is obtained bypairing spatially adjacent and different images of the plurality ofsamples.
 3. The method of claim 1, further comprising: (a) automaticallyidentifying a coordinate of the candidate object or area of interest inthe reconstructed image; and (b) automatically acquiring a selectedimage of the object or area of interest, at the location coordinatesobtained from the reconstructed image, wherein the selected imageincludes image information from two different images obtained at twodifferent coordinates during the scanning.
 4. The method of claim 1,wherein the candidate object or area of interest is detected byimmunohistochemistry.
 5. The method of claim 1, wherein the candidateobject or area of interest is detected by in situ hybridization.
 6. Themethod of claim 1, wherein the candidate object or area of interest isdetected by a stain.
 7. The method of claim 6, wherein the stain is anucleic acid dye selected from the group consisting of hematoxylin,Giemsa stain, methyl green, Nuclear Fast-Red, Hoechst 33342, Hoechst33258, thiazole orange, DAPI, ethidium bromide, propidium iodide, TOTO,YOYO-1, SYTOX Blue, SYTOX Green, 7-Aminoactinomycin,9-Amimo-6-chloro-2-methoxyacridine, and acridine homodimer.
 8. Themethod of claim 6, wherein the object or area of interest is stainedwith a cytoplasmic dye including at least one of eosin orKleihauer-Betke cytochemical stain or a combination thereof.
 9. Themethod of claim 1, wherein the candidate object or area of interest is acell specific marker.
 10. The method of claim 9 wherein the cellspecific marker is detected by a nuclear stain and counterstain.
 11. Themethod of claim 9, wherein the cell specific marker is detected byimmunohistochemistry, in situ hybridization, staining or a combinationthereof.
 12. The method of claim 1, wherein the reconstructed image is adigital image.
 13. The method as in claim 1, further comprising:operating an imaging system at a first magnification to obtain a firstset of the different images and a first reconstructed image; setting theimaging system at a second magnification greater than the firstmagnification to obtain a second set of the different images and asecond reconstructed image; using the first reconstructed image toidentify a target therein; and using the second reconstructed image toextract desired information about the target.
 14. A computer program,residing on a computer-readable medium, for automated image analysis ofa plurality of biological samples, comprising instructions to cause anautomated system to: (a) scan a plurality of biological samples at aplurality of different positions with different coordinates; (b) obtainan image at each of said different coordinates to produce differentimages from different locations; (d) reconstruct an image of theplurality of samples from the obtained different images comprisingpairing consecutive-corresponding images such that a reconstructed imageof the plurality of samples is obtained; and (e) process thereconstructed image to automatically identify a candidate object or areaof interest.
 15. The computer program of claim 14, wherein thereconstructed image of the plurality of sample is a digital image.